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Evaluation  of  the  ACEC 
Benchmark  Suite  for  Real-time  Applications 


Abstract:  This  technical  report  has  been  developed  by  the  Center  for  Software 
Engineering,  US  Army  CECOM  to  evaluate  the  Ada  Compiler  Evaluation  Capability 
(ACEC)  Version  1.0  benchmark  suite  for  measuring  the  performance  of  Ada 
compilers  meant  for  programming  real-time  systems.  The  ACEC  benchmarks  have 
been  analyzed  extensively  with  respect  to  their  measuring  of  Ada  real-time  features 
such  as  tasking,  memory  management,  input/output,  scheduling  and  delay  statement. 
Chapter  13  features,  pragmas,  interrupt  handling,  subprogram  overhead,  numeric 
computations  etc.  For  each  of  the  features  that  have  been  analyzed,  additional 
benchmarks  have  been  proposed.  Finally,  the  ACEC  benchmarks  that  measure  Ada 
features  important  for  programming  real-time  systems  have  been  run  on  two  Ada 
compilers,  namely  the  HP  Ada  compiler  self-hosted  on  HP  900/350  and  the  Verdix 
Ada  compiler  hosted  on  the  Sun  3/60  and  targeted  to  a  Motorola  68020  bare  machine 
and  their  results  listed. 
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1.  Introduction 


This  technical  report  has  been  developed  by  the  Center  for  Software  Engineering,  US 
Army  CECOM  to  evaluate  the  Ada  Compiler  Evaluation  Capability  (ACEC) 
benchmark  suite  (developed  by  Boeing  Aerospace  under  contract  to  the  US  Air  Force 
[1],  [2])  for  measuring  the  performance  of  Ada  compilers  meant  for  programming 
real-time  systems.  In  this  report,  the  ACEC  benchmarks  have  been  analyzed  for  their 
coverage  and  suitability  in  measuring  ihe  runtime  performance  of  Ada  features  that 
are  important  for  real-time  applications.  The  emphasis  of  the  ACEC  benchmarks  is 
on  measuring  the  runtime  performance  of  Ada  compilers,  although  compilation  issues 
have  also  been  addressed. 

The  principal  goal  of  Ada  is  to  provide  a  language  supporting  modem  software 
engineering  principles  in  the  design  and  development  of  real-time  systems.  To  design 
and  implement  real-time  systems,  it  is  essential  that  the  performance  as  well  as 
implementation  characteristics  of  an  Ada  compiler  system  meet  the  requirements  of  a 
real-time  application.  Many  current  Ada  implementations  do  not  allow  the 
development  of  reliable  embedded  systems  software  without  sacrificing  productivity 
and  quality.  One  of  the  reasons  for  the  above  is  that  real-time  programmers  have  no 
control  on  the  design  and  implementation  of  the  Ada  Runtime  System  (RTS)  except 
that  the  RTS  satisfy  the  requirements  listed  in  the  Ada  Language  Reference  Manual 
(LRM).  Due  to  the  effect  on  program  efficiency  and  reliability  of  the  various  runtime 
implementation  options,  simply  adopting  a  compiler  that  implements  the  language  as 
defined  in  the  LRM  is  insufficient  for  real-time  systems.  Benchmarks  are  needed  to 
determine  the  performance  as  well  as  implementation  strategies  of  various  Ada 
language  and  runtime  features  in  order  to  assess  a  compiler’s  suitability  for 
programming  real-time  systems. 

1.1  Objective 

The  CECOM  Center  for  Software  Engineering,  US  Army,  Fort  Monmouth  has  been 
involved  with  developing  benchmarks  for  Ada  language  and  runtime  system  features 
considered  important  for  programming  real-time  applications.  The  first  step  in  this 
effort  involved  the  identification  of  Ada  features  of  interest  for  real-time  systems.  The 
real-time  systems  analyzed  were  the  COMINT /ELINT  (Communication/Electronic 
Intelligence)  class  of  IEW  (Intelligence/Electronic  Warfare)  systems  supported  by  the 
US  Army.  A  side  result  of  this  effort  was  the  description  of  a  composite  benchmark 
for  COMINT /ELINT  class  of  IEW  systems  [3]. 

The  next  step  involved  the  development  of  real-time  benchmarks  that  measure  the 
Ada  features  identified  in  the  previous  effort.  Benchmarks  were  developed  that  a) 
measure  the  performance  of  Ada  individual  features,  b)  determine  Ada  runtime 
system  implementation  dependencies,  and  c)  test  algorithms  used  in  programming 
real-time  systems  [4], 
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As  part  of  this  ongoing  effort,  existing  benchmark  suites  were '  also  analyzed  to 
determine  their  suitability  for  evaluating  Ada  compiler  systems  for  real-time 
applications.  The  benchmark  suites  that  were  analyzed  included 

•  PIWG  Benchmarks:  developed  by  the  ACM  Performance  Issues  Working  Group 
[5]. 

•  University  of  Michigan  Benchmarks:  developed  at  the  University  of  Michigan  [6], 

•  Ada  Compiler  Evaluation  Capability:  developed  by  Boeing  Aerospace  for  the  US 
Air  Force  [1],  [2]. 

This  report  presents  the  results  of  the  analysis  of  ACEC  benchmarks  for  measuring 
the  runtime  performance  of  Ada  features  important  for  programming  real-time 
applications.  The  benchmarks  have  been  analyzed  with  respect  to: 

•  support  for  Ada  features  important  for  programming  real-time  applications 

•  interpretation  of  the  results  produced  by  running  the  benchmarks 

•  and  portability  of  the  ACEC  benchmarks.  •  . 

The  results  of  running  a  few  selected  ACEC  benchmarks  (that  measure  real-time  Ada 
features)  on  two  Ada  compilers,  namely  the  HP  Ada  Ada  compiler  which  is  self- 
hosted  on  a  HP  9000/350  computer  running  HP-UX  and  a  Verdix  cross-compiler 
hosted  on  a  Sun  3/60  and  targeted  to  a  Motorola  68020  bare  machine  are  also 
presented. 

1.2  Report  Layout 

This  report  is  divided  in  the  following  sections: 

Section  2  presents  a  brief  description  of  the  ACEC  benchmarks. 

Section  3  describes  the  typical  requirements  of  real-time  systems  and  correlates  them 
with  Ada  features  that  address  those  requirements.  It  also  discusses  the  criteria  for 
analysis  of  the  ACEC  benchmarks. 

In  Section  4,  the  ACEC  benchmarks  are  analyzed  with  respect  to  the  Ada  features 
identified  in  Section  3.  Section  4  also  comments  on  the  portability  of  the  benchmarks. 

Finally,  Section  5  concludes  with  some  thoughts  about  the  ACEC  benchmarks. 

Appendix  A  consists  of  a  list  of  tables  that  describe  the  ACEC  real-time  benchmarks. 
It  also  lists  the  results  of  running  the  ACEC  real-time  benchmarks  on  the  HP  and 
Verdix  Ada  compiler  systems. 

Appendix  B  comments  on  the  usefulness  of  the  ACEC  benchmarks  in  evaluating  Ada 
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compilers  from  a  real-time  perspective. 
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2.  Ada  Compiler  Evaluation  Capability  (ACEC)  Suite 


The  ACEC  benchmarks  were  developed  by  Boeing  Aerospace  Company  under 
contract  to  the  US  Air  Force.  The  ACEC  is  organized  as  a  set  of  essentially 
independent  test  problems  and  analysis  tools.  The  major  emphasis  of  the  ACEC  is  on 
execution  performance.  To  a  lesser  degree,  the  ACEC  will  also  test  for  compilation 
speed,  existence  of  language  features,  and  capacity.  The  tests  are  designed  to: 

•  Produce  quantitative  results,  rather  than  subjective  evaluations 

•  Be  as  portable  as  possible 

•  Require  minimal  operator  interaction 

•  Be  comparable  between  systems,  so  that  a  problem  run  on  one  system  can  be 
directly  compared  with  that  problem  run  on  another  target. 

The  ACEC  does  not  address  issues  such  as:  cost,  diagnostics  and  error  handling, 
adaptability  to  a  special  environment,  presence  of  support  tools,  and  target  processors 
such  as  vector  processors,  VOW  machine  architectures,  RISC  processors, 
multicomputers. 

The  ACEC  contains  a  large  number  of  test  problems  ('  1000).  Most  individual 
problems  are  fairly  small.  Many  address  one  language  feature  or  present  an  example 
which  is  particularly  well  suited  to  the  application  of  a  specific  optimization  technique. 

The  primary  focus  of  the  ACEC  is  on  comparing  performance  data  between  different 
compilation  systems  rather  than  on  studying  the  results  of  one  particular  system.  The 
analysis  tool  MEDIAN  computes  overall  relative  performance  factors  between 
systems  and  isolates  test  problems  where  any  individual  system  is  much  slower  (or 
faster)  than  expected,  relative  to  the  average  performance  of  all  systems  on  that 
problem  and  the  average  performance  of  the  problems  on  all  systems.  ACEC  users 
can  review  the  MEDIAN  report  to  isolate  the  strong  and  weak  points  of  an 
implementation  by  looking  for  common  threads  among  test  problems  which  report 
exceptional  performance  data.  The  ACEC  comparative  analysis  programs  compare 
performance  data  between  systems  and  identify  the  test  problems  which  show 
statistically  unusual  results.  The  results  of  some  test  problems  are  of  independent 
interest  -  such  as  rendezvous  times,  exception  propagation  time,  and  procedure  call 
time. 

The  ACEC  addresses  the  following: 

1.  Execution  Time  Efficiency 

2.  Code  Size  Efficiency 

3.  Compile  Time  Efficiency  (to  a  much  lesser  degree). 
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2.1  Execution  Time  Efficiency 

These  benchmarks  address  the  execution  speed  of  various  Ada  features  as  well  as 
those  aspects  of  an  Ada  Runtime  System  which  traditionally  have  been  the  province  of 
operating  systems.  These  benchmarks  have  been  subdivided  into  the  following  major 
categories: 

1.  Benchmarks  that  deal  with  Ada  Runtime  System  features 

2.  Benchmarks  that  measure  individual  Ada  language  features 

3.  Benchmarks  that  deal  with  performance  under  load 

4.  Benchmarks  that  deal  with  tradeoffs  between  performance  of  different  features 

5.  Benchmarks  that  deal  with  optimization  issues. 

6.  Application  Profile  benchmarks  that  are  further  subdivided  as  follows: 

•  Classical  Benchmark  Programs  (e.g.  Whetstone,  Dhrystone) 

•  Ada  in  Practice 

2.1.1  ACEC  Ada  Runtime  System  Benchmarks 

These  benchmarks  address  Ada  Runtime  System  Issues  that  traditionally  have  been 
the  domain  of  operating  systems  as  well  as  determining  runtime  implementation 
dependencies.  The  benchmarks  address  Ada  RTS  issues  such  as: 

1.  Tasking:  Tasking  benchmarks  can  be  further  subdivided  as  follows: 

a.  Task  Activation/Termination 

b.  Task  Synchronization 

c.  Exceptions  Raised  During  Rendezvous 

d.  Abort  Statement 

e.  Tasking  Runtime  Implementation  Dependencies 

f.  Tasking  Optimizations 

2.  Memory  Management:  Memory  management  benchmarks  have  been  subdivided 
into  two  areas: 

a.  Memory  Allocation  Timing  Benchmarks:  These  benchmarks  are  mainly 
tests  that  determine  timing  information  about  memory 
allocation/deallocation. 

b.  Memory  Allocation/Deallocation  Benchmarks:  These  benchmarks 
determine  the  way  storage  allocation/deallocation  is  implemented  for  a 
particular  Ada  compiler  system. 
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3.  Exception  Handling:  These  benchmarks  measure  the  time  to  raise,  propagate 
and  handle  exceptions. 

4.  Input /Output:  These  benchmarks  measure  time  for  input/output  operations  for 
TEXT  IO,  SEQUENTIALIO,  and  DIRECTIO.  Although  many  embedded 
targets  do  not  support  file  systems,  embedded  applications  may  make  intensive 
use  of  file  systems  and  the  performance  of  I/O  operations  is  critical  to  then- 
application  performance. 

5.  CLOCK  Function:  These  benchmarks  determine  the  overhead  due  to  the 
CLOCK  and  SECONDS  function. 

6.  Chapter  13  benchmarks:  These  benchmarks  measure  the  performance  of  various 
Chapter  13  features  such  as  Pragma  PACK,  SIZE  Representation  Clause, 
Record  representation  clause,  and  Unchecked  Conversion. 

7.  Scheduling  and  Delay  Statement:  These  benchmarks  determine  the  scheduling 
algorithms  and  the  impact  of  the  delay  statement. 

8.  Pragmas:  There  are  certain  predefined  pragmas  which  are  expected  to  have  an 
impact  on  the  execution  time  and  space  of  a  program.  These  include:  Pragmas 
CONTROLLED,  INLINE,  OPTIMIZE,  PACK,  PRIORITY,  SHARED,  and 
SUPPRESS.  There  are  test  problems  which  explore  the  performance  effects  of 
specifying  the  above  pragmas. 

2.1.2  ACEC  Individual  Language  Feature  Benchmarks 

There  are  test  problems  for  all  major  Ada  language  features.  The  test  suite  contains 
sets  of  test  problems  which  present  constructions  in  different  contexts.  The  results  will 
demonstrate  the  range  of  performance  associated  with  a  language  feature. 
Benchmarks  that  measure  individual  language  features  are  divided  into  the  following 
categories: 

1.  Record  object  manipulation 

2  Array  object  manipulation 

3.  Integer  Computations 

4.  Floating  point  computations 

5.  Fixed  point  computations 

6.  Loop  operations 

7.  Constraint  checking 

8.  Type  conversions  from  one  type  to  another 

9.  Mathematical  functions 

10.  Subprogram  Overhead  {including  Generics) 
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2.13  Performance  Under  Load 

There  are  some  language  constructions  which  display  nonuniform  performance.  The 
more  of  them  in  a  program,  the  slower  the  average  performance.  These  are  tests  that 
determine  the  performance  of  such  language  constructions  under  different  loading 
scenarios.  Examples  include  task  loading,  levels  of  nesting,  parameter  variation,  and 
declarations. 

2.1.4  Tradeoffs 

In  many  areas  of  the  language,  it  is  possible  to  speed  up  the  performance  of  one 
feature  at  the  cost  of  slowing  another  down.  Areas  that  have  been  covered  include 
design  issues,  and  context  variation. 

2.1.5  Optimizations 

Specific  optimization  test  problems  include  examples  where  it  is  easy  and  where  it  is 
more  difficult  to  determine  that  the  optimization  is  applicable.  There  are  some  test 
problems  which  perform  the  same  basic  operations,  but  have  a  modification  which 
either  performs  the  intended  optimization  in  the  source  text,  or  precludes  the 
application  of  the  optimization.  Optimizations  that  have  been  addressed  include: 

1.  Common  Subexpression  Elimination 

2.  Folding 

3.  Loop  Invariant  Motion 

4.  Strength  Reduction 

5.  Dead  Code  Elimination 

6.  Register  Allocation 

7.  Loop  Interchange 

8.  Loop  fusion 

9.  T<  st  Merging 

10.  Boolean  Expression  Optimization 

11.  Algebraic  Simplification 

12.  Order  of  Expression  Evaluation 

13.  Jump  tracing 

14.  Unreachable  Code  Elimination 
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15.  Use  of  Machine  Idioms 

2.1.6  Application  Profile  Tests 

The  ACEC  also  includes  examples  from  actual  application  code.  This  code  contains 
test  problems  representative  of  how  Ada  is  being  used  in  practice.  These  are  example 
test  problems  drawn  from  Ada  programs  extracted  from  projects.  Examples  include 
Avionics  Applications,  Electronic  Warfare  Application  feasibility  study  code,  and 
Radar  Application  code. 

Application  profile  tests  have  been  subdivided  as  follows: 

1.  Classical  Tests:  The  ACEC  test  suite  contains  classical  benchmark  programs 
coded  in  Ada.  Classical  tests  include:  Ackermann’s  function,  Kalman  filter. 
Autocorrelation  program,  Quicksort  and  variation  of  quicksorts,  Mergesort, 
Dhrystone  and  Whetstone  benchmarks,  and  Gamm  measure  benchmark. 

2.  Ada  in  Practice:  These  are  example  test  problems  drawn  from  Ada  programs 
extracted  from  projects.  They  represent  typical  usage  of  Ada. 

2.2  Code  Size  Efficiency 

The  memory  size  of  programs  is  an  important  attribute  in  many  mission  critical 
applications.  On  embedded  systems,  memory  is  often  a  limited  resource.  On  some 
target  processors  such  as  the  MIL-STD-1750A,  while  physical  memory  may  be 
available,  maintaining  addressability  is  critical  and  a  small  code  expansion  rate  can 
help  system  design  by  reducing  the  need  to  switch  memory  states.  There  are  two  size 
measurements  of  most  interest  to  Ada  projects:  the  amount  of  space  generated  inline 
to  translate  each  statement  (Code  Expansion  Size),  and  the  amount  of  space  occupied 
by  the  RTS  (Rimtime  System  Size). 

Code  Expansion  Size:  The  code  expansion  size  is  measured  in  the  timing  loop.  It  is  the 
space  in  bits,  between  the  beginning  and  end  of  each  test  problem.  In  this  report,  the 
code  expansion  size  of  problems  have  been  considered  in  light  of  their  ability  to  help 
interpret  results. 

Runtime  System  Size:  The  size  of  the  Runtime  System  is  an  important  parameter  to 
many  projects.  Space  taken  by  the  RTS  is  not  available  for  use  by  application  code,  so 
a  small  RTS  will  permit  larger  applications  to  be  developed. 

23  Compile  Time  Efficiency 

The  times  to  compile  the  compilation  units  are  collected  and  analyzed.  The 
benchmarks  were  developed  to  measure  execution  time  performance  aspects,  and  do 
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not  necessarily  represent  a  set  of  compilation  units  which  will  expose  all  the  relevant 
compilation  time  variables.  However,  they  do  represent  a  set  of  programs  which  will 
exercise  a  compiler  and  observing  the  compile  time  of  these  programs  can  give  insight 
to  the  overall  compilation  rates. 

2.4  ACEC  Summary 

The  philosophy  of  the  ACEC  is  that  end  users  will  not  have  to  examine  in  detail  each 
individual  test  problem.  Rather,  they  should  run  the  test  suite  and  let  the  analysis  tools 
isolate  problems  where  a  system  does  unusually  well  or  unusually  poorly.  These 
problems  can  then  be  examined  in  more  detail  to  try  to  determine  what  characteristics 
of  the  problem  are  responsible  for  the  unusual  behavior. 

More  information  about  the  ACEC  benchmarks  can  be  obtained  from  references  [1] 
and  [2]  as  well  as  from: 

Raymond  Szymanski 
WRDC/AAAF-3 
Wright-Patterson  AFB 
Ohio  45433-6543 
(513)  255-3947 


3.  Real-time  Systems  and  Ada  Benchmarking 

Before  jumping  into  the  ana. /sis  of  ACEC  benchmarks,  it  is  important  to  understand 
the  typical  software  requirements  of  real-time  applications,  how  Ada  addresses  those 
requirements  and  the  issues  involved  in  benchmarking  Ada  features  that  address  these 
requirements. 

3.1  Requirements  Of  Real-time  Systems 

For  a  programming  language  to  be  used  effectively  to  program  real-time  embedded 
systems,  it  should  be  able  to  support  the  following  characteristics  (for  more  details  see 
reference  [4]): 

•  Real-time  Preemptive  Scheduling 

•  Concurrency,  Inter-task  and  Intra-task  communication 

•  Time  Abstraction 

•  Interaction  with  Real  World 

•  Input /Output 

•  Resource  Utilization 

•  Numeric  Computations 

•  Fault  Tolerance 

•  Event-driven  Reconfiguration 

•  Reliability 

3.2  Ada  and  Real-time  Requirements 

Programmers  generally  have  no  control  on  the  design  and  implementation  of  the  Ada 
runtime  system  except  that  it  satisfy  the  requirements  listed  in  the  LRM.  Table  1  lists 
the  Ada  features  that  support  real-time  requirements. 
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From  Table  1,  it  is  clear  that  a  real-time  benchmarking  suite  should  address  the 
following  real-time  Ada  features: 

1.  Tasking 

2  Memory  management 

3.  Exceptions 

4.  Input/Output 

5.  Clock  Function 

6.  Scheduling  and  Delay  Statement 

7.  Chapter  13  Benchmarks 

8.  Interrupt  Handling 

9.  Pragmas 

10.  Subprogram  Overhead 

11.  Numeric  Computations 

3.3  Ada  Benchmarking 

Ada  benchmarking  can  be  approached  in  4  ways: 

1.  Benchmarks  that  measure  execution  speed  of  individual  features  of  the 
language. 

2.  Benchmarks  that  determine  implementation  dependent  attributes. 

3.  Benchmarks  that  measure  the  performance  of  commonly  used  real-time  Ada 
paradigms  (that  may  be  programmed  using  macro  constructs  [4]). 

4.  Composite  benchmarks  which  include  representative  code  from  real-time 
applications. 

A  detailed  description  of  benchmarking  approaches  is  presented  in  the  report  titled 
"Real-time  Performance  Benchmarks  For  Ada"  [4]. 
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4.  Analysis  of  the  ACEC  Benchmarks 

The  ACEC  benchmarks  have  been  analyzed  with  respect  to  the  following 
characteristics: 

1.  Features  measured  by  the  Benchmarks 

The  ACEC  suite  has  been  analyzed  with  respect  to  its  ability  to  measure  the 
performance  and  to  determine  the  implementation  characteristics  of  Ada 
features  that  are  important  for  programming  real-time  systems.  The  ACEC 
tests  are  examined  with  respect  to  each  of  the  following  real-time  features  [4]: 

•  Tasking 

•  Memory  management 

•  Exceptions 

•  Input/Output 

•  Clock  Function 

•  Scheduling  and  Delay  Statement 

•  Chapter  13  Benchmarks 

•  Interrupt  Handling 

•  Pragmas 

•  Subprogram  Overhead 

•  Numeric  Computations 

In  addition  to  their  measurement  of  individual  Ada  features,  the  ACEC  suite 
has  also  been  evaluated  with  respect  to  its  implementation  of  real-time 
paradigms  and  composite  benchmarks. 

2.  Information  provided  for  interpretation  of  the  results 

Running  the  ACEC  benchmarks  produces  a  set  of  numbers  which  have  to  be 
interpreted.  It  is  important  that  the  benchmarking  suite  provide  sufficient 
information  about  interpreting  the  results. 

3.  Portability 

The  benchmarks  should  be  portable  and  executable  on  any  Ada  compiler  system 
with  minimum  modifications.  There  are  come  benchmarks  (like  interrupt 
handling)  that  may  not  be  portable  and  depend  on  the  hardware  being  tested. 
The  ACEC  suite  has  been  analyzed  with  respect  to  its  ease  of  portability  to 
various  Ada  compiler  systems. 
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The  ACEC  evaluation  format  is  as  follows: 

•  For  each  feature,  the  ACEC  benchmarks  which  address  that  feature  have  been 
identified  and  their  description  presented  in  tables  listed  in  Appendix  A.  Also 
listed  are  the  results  of  running  the  ACEC  It  also  presents  the  results  of  running 
the  ACEC  real-time  benchmarks  on  the  following  Ada  compilers:  HP-Ada 
Compiler  (Releases  3.25  and  4.35)  running  on  HP  9000/350  machine  under  HP- 
UX  Release  6.2;  and  Verdix  Ada  Compiler  (Release  5.41)  hosted  on  a  Sun  3/60 
and  targeted  to  a  Motorola  68020  bare  machine. 

•  Then,  comments  are  presented  on  those  set  of  benchmarks.  The  comments 
address  two  major  areas,  namely  any  deficiencies  in  a)  the  benchmarks  themselves 
and  b)  additional  information  that  is  not  provided  by  the  ACEC  in  interpretation 
of  the  results  produced  by  running  those  benchmarks. 

•  Finally,  additional  benchmarks  not  covered  in  the  ACEC  are  listed  when 
appropriate  for  the  feature  analyzed. 

4.1  Tasking 

For  the  purposes  of  this  discussion,  the  ACEC  benchmarks  have  been  analyzed  with 
respect  to  their  measurement  of  the  following  aspects  of  tasking: 

•  Task  Activation/Termination 

•  Task  Synchronization 

•  Exceptions  Raised  During  Rendezvous 

•  Abort  Statement 

•  Tasking  Priorities 

•  Miscellaneous  Tasking  Benchmarks 

•  Tasking  Optimizations 

4.1.1  Task  Activation/Termination 

Task  Activation/Termination  is  an  important  benchmark  for  real-time  systems.  Task 
elaboration,  activation  and  termination  are  almost  always  suspect  operations  in  real¬ 
time  programming  and  programmers  often  allocate  tasks  statically  to  avoid  runtime 
execution  time. 

Table  2  lists  the  ACEC  benchmarks  for  measuring  task  activation/termination  timings 
along  with  the  results  of  running  these  benchmarks  on  the  HP  and  Verdix  Ada 
compilers.  The  task  to  be  activated  can  either  be  an  object  of  a  task  type  or  can  be 
activated  using  the  new  allocator.  The  difference  in  the  times  provided  by  these  tests 
give  some  insight  into  the  relative  efficiency  of  the  two  types  of  task  activation. 
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Comments:  Observations  about  the  ACEC  task  activation/termination  benchmarks 
are: 

1.  The  tasks  whose  activation/termination  times  are  being  measured  are  very 
simple  tasks  which  have  a  null  statement  inside  the  task  body.  This  is  not  a  very 
realistic  scenario  as  it  is  quite  possible  that  many  compilers  realizing  that  this 
task  does  nothing  may  optimize  it  away  and  the  measurements  obtained  may  not 
be  correct.  The  task  body  should  do  something  meaningful  such  as  call  a 
subprogram  that  performs  some  meaningful  calculations. 

2.  The  time  to  elaborate,  activate  and  terminate  a  task  is  measured  as  one  value. 
The  individual  components  of  the  measurements  are  too  quick  to  measure  with 
the  available  CLOCK  resolution. 

3.  An  important  criteria  for  tasking  benchmarks  is  the  STORAGESIZE  used  by 
the  tasks  that  are  elaborated.  Some  implementations  may  implicitly  deallocate 
the  task  storage  space  on  return  from  a  procedure  or  on  exit  from  a  block 
statement  (when  the  task  object  is  declared  in  that  procedure  or  block 
statement).  If  task  space  is  implicitly  deallocated,  the  number  of  iterations  can 
be  increased  to  get  greater  accuracy  for  task  activation/termination 
measurement.  So  if  task  space  is  not  deallocated  on  return  from  a  procedure  or 
block  statement,  TASKTYPE’STORAGESIZE  can  be  changed  such  that  the 
number  of  iterations  can  be  increased  (thus  increasing  the  accuracy  of  the 
measurement). 

Additional  Task  Activation/Termination  Benchmarks: 

1.  More  task  activation/termination  benchmarks  are  needed  to  determine  if  a 
real-time  programmer  can  declare  tasks  for  time-critical  modules  in  a)  a  block 
statement  or  b)  within  other  tasks. 

Benchmark:  Measure  task  activation  and  termination  time  ( without  the  new 
operator)  where 

•  Task  type  is  declared  in  the  mean  program  and  task  object  is  declared  in  a 
block  statement  in  the  main  program. 

•  Task  type  and  task  object  are  declared  in  another  task  which  is  declared  in  the 
main  program. 

2.  Task  activation/termination  times  may  degrade  as  the  number  of  active  tasks  in 
the  system  increases.  This  is  a  more  realistic  scenario  for  a  real-time  system  as 
generally  there  are  existing  tasks  when  new  tasks  are  activated.  As  more  and 
more  tasks  are  created,  task  activation  time  may  increase  due  to  the  possible 
increase  in  storage  allocation  time. 

Benchmark:  Measure  the  affect  on  task  activation/termination  times  as  the 
number  of  existing  active  tasks  keeps  on  increasing. 

Number  of  existing  active  tasks  in  the  system  could  vary  from  1  to  20. 
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3.  During  the  execution  of  an  Ada  program,  a  low  priority  task  spawns  a  task. 
While  the  activation  of  this  spawned  task  is  occurring,  if  a  high  priority  task 
becomes  ready  to  execute,  it  may  remain  suspended  until  the  completion  of  the 
low  priority  task  activation. 

Benchmark :  Determine  if  a  low  priority  task  activation  could  result  in  a  very  long 
suspension  of  a  high  priority  task. 

4.1.2  Task  Synchronization 

In  Ada,  tasks  communicate  with  each  other  via  the  rendezvous  mechanism. 
Rendezvous  are  effectively  similar  to  procedure  calls,  yet  they  are  much  more  complex 
to  implement,  and  therefore  create  a  tremendous  amount  of  overhead  for  the  runtime 
system.  Because  of  the  timing  constraints  in  a  real-time  system,  it  is  essential  that  the 
rendezvous  mechanism  be  as  efficient  as  possible. 

The  ACEC  suite  has  a  comprehensive  set  of  task  synchronization  benchmarks.  These 
benchmarks  are  divided  in  the  following  logical  areas. 


4.1.2.1  Time  For  a  Simple  Rendezvous 

These  benchmarks  measure  the  time  for  a  simple  rendezvous  and  no  parameters  are 
passed  during  the  rendezvous.  Time  is  measured  to  complete  a  rendezvous  between  a 
task  and  a  procedure  with  no  additional  load  present.  This  method,  then  gives  a  lower 
bound  on  rendezvous  time,  because  no  extraneous  units  of  execution  are  competing 
for  the  CPU.  Table  3  lists  the  simple  rendezvous  benchmarks  and  Table  4  lists  the 
benchmarks  that  determine  rendezvous  performance  with  varying  number  of  tasks. 
The  results  of  running  these  benchmarks  on  the  HP  and  Verdix  Ada  compilers  are 
also  listed. 

Comments:  Comments  about  ACEC  simple  rendezvous  benchmarks  are: 

1.  The  ACEC  benchmarks  measure  simple  rendezvous  timings  for  rendezvous 
between  equal  priority  tasks,  as  well  as  rendezvous  between  tasks  of  different 
priorities.  Two  context  switches  are  required  for  rendezvous  between  tasks  of 
different  priorities  as  opposed  to  a  single  context  switch  for  rendezvous  between 
tasks  of  same  priority.  So  any  deviation  from  the  expected  results  points  to  a 
bad  compiler  implementation. 

2.  Rendezvous  times  with  tasks  in  subunits  as  well  as  tasks  in  separate  packages 
should  be  the  same  as  if  those  tasks  were  in  the  main  program.  A  task  being  in  a 
subunit  or  a  separate  package  should  affect  compilation  times  and  not  execution 
times. 
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4.1 22  Select  Statement  With  Else  Alternative 

These  benchmarks  measure  the  time  it  takes  to  execute  a  select  statement  with  an  else 
alternative  under  various  scenarios.  The  ELSE  alternative  is  always  executed.  Table 
5  lists  these  benchmarks  along  with  the  results  of  running  these  benchmarks  on  the 
HP  and  Verdix  Ada  compilers. 

Comments:  Observations  about  these  benchmarks  are: 

1.  All  of  these  benchmarks  have  a  select  statement  with  an  else  alternative  (which 
is  always  executed).  Hence,  execution  of  these  benchmarks  will  enable  real¬ 
time  programmers  to  determine  if  a  compiler  optimizes  away  the  entry  call  or 
accept  statement  and  directly  executes  the  else  alternative. 

4. 1.2.3  Rendezvous  Calls  with  Conditional  Selects 

These  benchmarks  have  conditional  select  statements  with  various  scenarios.  The 
tests  are  listed  in  Table  6  along  with  the  results  of  running  these  benchmarks  on  the 
HP  and  Verdix  Ada  compilers. 

Comments:  Observations  about  these  benchmarks  are: 

1.  Benchmarks  (taskl6)  that  contain  delay  statements  with  a  negative  argument  do 
not  serve  any  useful  purpose  for  real-time  programmers,  as  negative  delay 
statements  are  never  used  in  real-time  systems. 

2.  There  is  no  information  provided  on  interpretation  of  the  results  produced  by 
executing  benchmarks  taskl5,  taskl6,  taskl7,  task21,  and  task22.  Upon  further 
analysis,  it  is  determined  that  these  benchmarks  do  not  provide  any  useful 
information  in  evaluating  an  Ada  compiler. 

4. 1.2.4  Selective  Wait 

These  benchmarks  test  various  scenarios  with  the  selective  wait  statement.  Table  7 
lists  the  selective  wait  benchmarks  along  with  the  results  of  running  these  benchmarks 
on  the  HP  and  Verdix  Ada  compilers. 

Comments:  Observations  about  these  benchmarks  are: 

1.  Benchmark  (task34)  measures  the  time  to  check  the  entry  queue  for  a  accept 
statement.  Depending  on  the  compiler  implementation,  time  to  check  the  entry 
queue  could  force  missed  deadlines  in  real-time  systems. 

2.  Benchmarks  (task35)  that  contain  delay  statements  with  a  0.0  argument  do  not 
serve  any  useful  purpose  for  real-time  programmers,  as  delay  statements  with 
0.0  argument  are  never  used  in  real-time  systems. 
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3.  In  benchmark  task59,  all  delay  alternatives  are  negative  in  the  selective  wait.  As 
pointed  out  before,  negative  delay  alternatives  do  not  provide  any  useful 
information  in  Ada  compiler  evaluation  for  real-time  programming. 

Additional  Task  Synchronization  Benchmarks: 

1.  For  some  compilers,  the  more  the  number  of  entries  in  a  select  statement,  the 
more  time  it  takes  to  rendezvous  with  any  entry  in  the  select  statement.  For 
some  implementations,  time  for  a  rendezvous  may  also  be  affected  by  the 
position  of  the  accept  alternative  in  the  select  statement. 

Benchmark :  Measure  the  effect  on  rendezvous  time  as  the  number  of  accept 
alternatives  in  a  select  statement  increases. 

Main  program  calls  first  (middle,  last)  entrv  r  select  statement  in  another 
task  as  the  number  of  accept  statements  increases  from  2  to  10  to  20. 

2.  Rendezvous  time  may  depend  on  the  number  of  open  guard  statements.  For 
some  implementations,  rendezvous  time  may  depend  on  the  number  of  guards 
in  the  select  statement  and  the  position  of  the  accept  in  the  select  statement. 

Benchmark:  Measure  the  effect  of  guards  on  rendezvous  time,  where  the  main 
program  calls  an  entry  in  another  task  as  the  number  of  accept  alternatives  in  the 
select  statement  increases. 

3.  Rendezvous  time  may  depend  on  the  size  and  type  of  the  passed  parameters 
which  may  involve  both  the  task  stacks  or  the  allocation  of  a  separate  area  for 
passing  large  structures.  Increasing  rendezvous  times  for  array  parameters  as 
the  size  of  the  array  increases  implies  that  the  implementation  uses  pass  by  copy 

"tead  of  pass  by  reference. 

Benchmark:  Measure  the  time  required  for  a  complex  rendezvous,  where  a 
procedure  in  the  main  program  calls  an  entry  in  another  task  with  different  type, 
number  and  mode  of  the  parameters. 

The  types  of  the  parameters  include  a)  integer  arrays  (size  1  to  1000  to  10000), 
and  b)  1  to  100  integers.  The  mode  of  the  parameters  passed  is  either  out  or  in 
out. 

4.  Fairness  of  select-alternative  is  a  particular  aspect  of  scheduling  fairness.  If  a 
task  reaches  a  selective  wait  and  there  is  an  entry  call  waiting  at  more  than  one 
open  alternative,  or  if  a  task  is  waiting  at  a  selective  wait  and  more  than  one 
open  accept  or  delay  alternative  becomes  eligible  for  selection  at  the  same  time, 
an  alternative  is  selected  according  to  criteria  that  are  not  specified  in  the  LRM. 
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Benchmar/c  Determine  algorithm  used  when  choosing  among  branches  of  a 
selective  wait  statement. 


5.  The  order  in  which  an  Ada  compiler  system  chooses  to  evaluate  the  guard 
conditions  in  a  select  statement  is  implementation  dependent.  Real-time 
programmers  may  need  to  know  the  order  in  which  the  guard  conditions  are 
evaluated. 

Benchmark :  Determine  the  order  of  evaluation  for  guard  conditions  in  a  selective 
wait 

4.13  Exceptions  During  a  Rendezvous 

If  an  exception  is  raised  within  a  rendezvous,  it  is  propagated  to  the  task  containing 
the  accept  as  well  as  to  the  calling  task.  This  is  the  most  complex  form  of  exception 
handling  since  the  exception  is  handled  in  both  the  task  containing  the  accept  and  the 
calling  task.  For  real-time  systems,  it  is  important  to  measure  the  time  it  takes  to 
handle  exceptions  raised  during  a  rendezvous.  Table  8  lists  the  ACEC  benchmarks  for 
exceptions  raised  during  a  rendezvous  along  with  the  results  of  running  these 
benchmarks  on  the  HP  and  Verdix  Ada  compilers. 

Comments: 

1.  ACEC  benchmarks  for  handling  exceptions  within  a  rendezvous  are  adequate 
and  do  not  require  any  additional  benchmarks. 

4.1.4  Abort  Statement 

Quick  restarts  of  tasks  are  required  in  a  number  of  real-time  embedded  systems.  Ada 
model  of  concurrency  does  not  provide  an  abstraction  where  a  task  may  be 
asynchronously  notified  that  it  must  change  its  current  execution  state.  One  way  to 
implement  asynchronous  change  in  control  is  to  abort  the  task  and  then  replace  it  with 
a  new  one.  Table  9  lists  the  tests  for  abort  statement  along  with  the  results  of  running 
these  benchmarks  on  the  HP  and  Verdix  Ada  compilers.  These  tests  measure  timings 
to  abort  tasks  under  various  scenarios. 

Comments:  None. 

Additional  Abort  Statement  Benchmarks: 

More  task  abortion  benchmarks  are  needed  as  follows: 

1.  In  real-time  systems,  tasks  may  have  to  be  aborted  in  a  certain  sequence.  The 
semantics  of  the  abort  statement  do  not  guarantee  immediate  completion  of  the 
named  task.  Completion  must  happen  no  later  than  when  the  task  reaches  a 
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synchronization  point. 

Benchmark :  Determine  order  of  evaluation  of  tasks  named  in  an  abort  statement. 


2.  When  a  task  has  been  aborted,  it  may  become  completed  at  any  point  from  the 
time  the  abort  statement  is  executed  until  its  next  synchronization  point. 
Depending  on  when  an  implementation  actually  causes  the  task  to  complete  the 
results  of  an  aborted  task  may  be  different.  Suppose  a  task  is  updating  a 
variable  that  is  visible  to  other  tasks,  prior  to  a  synchronization  point.  If  the  task 
is  aborted  just  prior  to  the  update,  it  may  leave  the  variable  unchanged  if  it 
becomes  completed  immediately,  or  it  may  update  the  variable  and  then 
becomes  completed  at  the  synchronization  point.  This  could  affect  the  results  of 
the  whole  program. 

Benchmark:  Determine  the  results  if  a  task  is  aborted  while  updating  a  variable  ? 

4.1.5  Task  Priorities 

The  ACEC  suite  has  several  tests  that  use  Pragma  PRIORITY  to  determine 
rendezvous  timings  under  different  scenarios.  These  tests  have  already  been  discussed 
in  Section  4.3.2  under  Task  Synchronizatioa  However,  additional  benchmarks  are 
needed  for  tasking  priorities. 

Additional  Tasking  Priority  Benchmarks: 

1.  Programmers  may  need  to  know  the  default  priority  of  the  main  program  and 
other  tasks  in  order  to  design  usable  embedded  systems. 

Benchmark:  Determine  priority  of  tasks  (and  of  the  main  program)  that  have  no 
defined  priority. 


2.  If  two  tasks  without  explicit  priorities  conduct  a  rendezvous,  and  if  the  priority 
given  to  the  rendezvous  is  higher  than  a  task  with  an  explicit  priority,  the  Ada 
program  may  perform  in  an  unpredictable  manner. 

Benchmark:  Determine  priority  of  a  rendezvous  between  two  tasks  without  explicit 
priorities. 

4.1.6  Miscellaneous  Tasking  Benchmarks 

There  are  some  tasking  benchmarks  that  do  not  fall  under  any  of  the  above  defined 
categories. 

Table  10  lists  the  miscellaneous  tasking  benchmarks  along  with  the  results  of  running 
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these  benchmarks  on  the  HP  and  Verdix  Ada  compilers. 

Comments:  Observations  about  these  benchmarks  are: 

1.  Benchmark  (task48)  is  not  very  useful  as  the  timing  of  interest  is  the  time  taken 
to  invoke  the  interrupt  handler  when  an  actual  hardware  interrupt  is  received  as 
opposed  to  calling  an  entry  tied  to  an  interrupt  directly. 

Additional  Miscellaneous  Benchmarks: 

1.  A  group  of  tasks  (children  of  the  same  parent)  can  terminate  by  using  the 
terminate  option  of  the  select  statement.  If  the  overhead  due  to  the  terminate 
option  is  high,  then  this  option  should  not  be  used  (especially  if  the  selective  wait 
is  inside  a  loop). 

Benchmark:  Measure  the  cost  of  using  the  terminate  option  in  a  select  statement 

2.  In  many  real-time  embedded  systems  where  space  is  at  a  premium  it  may  be 
desirable  that  task  space  be  deallocated  when  that  task  terminates. 

Benchmark:  Determine  if  task  space  is  deallocated  on  return  from  a  procedure 
when  a  task  that  has  been  allocated  via  the  new  operator  in  that  procedure 
terminates. 

3.  It  might  be  impossible  for  a  runtime  system  to  deallocate  the  task  storage  space 
after  termination.  This  is  because  the  access  value  might  have  been  copied  and 
an  object  might  still  be  referencing  the  terminated  task’s  task  control  block. 

Benchmark :  Determine  if  tasks  that  are  allocated  dynamically  by  the  execution  of 
an  new  allocator  do  not  have  their  space  reclaimed  upon  termination  when  access 
type  is  declared  in  a  library  unit  or  outermost  scope. 

4.  When  several  tasks  are  activated  in  parallel,  the  order  of  their  elaboration  may 
affect  program  execution. 

Benchmark :  Determine  the  order  of  elaboration  when  several  tasks  that  are 
declared  in  the  same  declarative  region  are  activated  in  parallel 

5.  The  activation  of  tasks  proceeds  in  parallel.  Correct  execution  of  a  program  may 
depend  on  a  task  continuing  execution  after  its  activation  is  completed  but 
before  all  other  tasks  activated  in  parallel  have  completed  their  respective 
activations. 

Benchmark :  Determine  if  a  task,  following  its  activation  but  prior  to  the  completion 
of  activation  of  tasks  declared  in  the  same  declarative  part,  continue  execution. 
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6.  The  LRM  does  not  define  when  STORAGEJERROR  must  be  raised  should  a 
task  object  exceed  the  storage  allocation  of  its  creator  or  master.  The  exception 
must  be  no  later  than  task  activation;  however  an  implementation  may  choose  to 
raise  it  earlier. 

Benchmark:  Determine  when  exception  is  raised  if  the  allocation  of  a  task  object 
raises  STORAGE  ERROR. 

7.  For  some  real-time  embedded  applications,  it  is  desirable  that  tasks  declared  in 
a  library  package  do  not  terminate  when  the  main  program  terminates.  System 
designers  may  need  to  know  this  information. 

Benchmark:  Determine  if  tasks  declared  in  a  library  package  terminate  when  the 
mean  program  terminates. 

4.1.7  Task  Optimization 

These  benchmarks  are  designed  to  determine  if  certain  tasking  optimizations  have 
been  implemented  by  Ada  compilers. 

Table  11  lists  the  ACEC  Habermann-Nassi  tasking  optimization  benchmarks  Table  12 
lists  the  other  tasking  optimization  benchmarks.  The  results  of  running  these 
benchmarks  on  the  HP  and  Verdix  Ada  compilers  are  also  listed. 

Comments: 

1.  As  far  as  tasking  optimizations  are  concerned,  the  ACEC  suite  is  quite 
comprehensive. 

4.2  Memory  Management 

Memory  management  benchmarks  have  been  divided  into  two  separate  areas: 

1.  Memory  Allocation  Timing  Benchmarks:  These  benchmarks  are  mainly  tests  that 
determine  timing  information  about  memory  allocation/deallocation. 

2.  Memory  Allocation (Deallocation  Benchmarks:  These  benchmarks  determine  the 
way  storage  allocation/deallocation  is  implemented  for  a  particular  Ada 
compiler  system. 

4.2.1  Memory  Allocation  Timing  Benchmarks 

Since  time  and  space  are  at  a  premium  in  real-time  embedded  systems,  it  is  essential 
that  the  dynamic  memory  allocation  and  deallocation  be  as  efficient  as  possible. 
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Real-time  programmers  need  to  know  the  maximum  time  to  allocate  and  deallocate 
storage  for  a  particular  Ada  compiler  in  order  to  ensure  that  performance 
requirements  will  be  met  for  their  application. 

Table  13  lists  the  ACEC  memory  allocation  timing  benchmarks  along  with  the  results 
of  running  these  benchmarks  on  the  HP  and  Verdix  Ada  compilers. 

Comments:  Observations  about  these  benchmarks  are: 

1.  In  benchmarks  (ss22,  ss23,  and  ss24),  time  to  to  allocate  small  one,  two,  and 
three  dimensional  arrays  of  float  is  measured.  The  sizes  of  the  arrays  are 
different  thus  limiting  the  usefulness  of  the  results  obtained  by  running  these 
benchmarks.  What  is  of  interest  is  the  time  for  allocating  same  size  array  as  the 
number  of  dimensions  increase. 

2.  Benchmark  (ss25),  which  measures  time  to  allocate  a  small  dynamically 
bounded  one  array  of  float,  does  not  provide  any  useful  information  as  the 
timing  has  to  be  compared  to  allocation  timings  for  same  size  and  multiple 
dimension  arrays. 

Additional  Memory  Allocation  Timing  Benchmarks: 

1.  Tests  for  more  Ada  types  are  needed  to  determine  their  allocation  overhead 
time.  Times  to  allocate  various  numbers  of  types  INTEGER  and 
ENUMERATION  have  to  be  measured  as  well  as  the  times  to  allocate  various 
sizes  of  arrays,  records,  and  STRINGS.  The  objective  is  to  determine  the 
allocation  overhead  involved  and  if  there  is  any  difference  in  the  overhead  based 
on  the  type  of  object  allocated. 

Benchmark:  Measure  time  for  allocating  storage  known  at  compile  time.  Times  to 
allocate  various  numbers  of  types  INTEGER  and  ENUMERATION  are 
measured  as  well  as  the  times  to  allocate  various  sizes  of  arrays,  records,  and 
STRINGS. 

2.  More  tests  are  needed  to  determine  if  allocation  time  is  dependent  on  size  (in 
the  composite-type  object  case).  Also,  based  on  these  timing  measurements 
real-time  programmers  can  decide  whether  to  use  the  new  allocator  for  object 
elaboration  or  to  declare  the  object  as  in  the  fixed  length  case. 

Benchmark :  Memory  Allocation  via  the  New  Allocator.  Allocation  time  of  objects 
of  type  INTEGER,  and  ENUMERATION  as  well  as  composite  type  objects  of 
various  sizes  are  measured. 

In  these  tests,  the  objects  that  have  been  allocated  via  the  new  allocator  have 
also  been  freed  via  Unchecked_Deallocation  before  exiting  the  scope  in  which 
the  object  was  allocated. 
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3.  If  memory  is  allocated  in  a  loop  via  the  new  allocator  and  the  memory  that  is 
allocated  is  not  freed  via  Unchecked_Deallocation,  then  the  time  required  for 
dynamic  memory  allocation  can  be  affected  as  more  space  is  allocated. 

Benchmark :  Determine  the  effect  on  time  required  for  dynamic  memory  allocation 
when  memory  is  continuously  allocated  without  being  freed  via 
Unchecked  Deallocation  in  the  scope  where  the  memory  was  allocated. 

4.2.2  Memory  Allocation/Deallocation  Benchmarks 

It  is  important  for  real-time  programmers  to  know  if  a  particular  compiler 
implementation 

•  deallocates  nothing 

•  supports  only  UNCHECKED  DEALLOCATION 

•  deallocates  all  the  storage  for  an  access  type  when  the  scope  of  the  access  type  is 
left 

•  detects  inaccessible  storage  and  automatically  deallocates  it  (garbage  collection). 

Table  14  lists  the  ACEC  memory  allocation/deallocation  benchmarks  along  with  the 
results  of  running  these  benchmarks  on  the  HP  and  Verdix  Ada  compilers. 

Comments : 

1.  ACEC  suite  is  very  comprehensive  in  memory  allocation/deallocation 
benchmarks. 

43  Exceptions 

Table  15  lists  the  ACEC  exception  handling  timing  benchmarks  along  with  the  results 
of  running  these  benchmarks  on  the  HP  and  Verdix  Ada  compilers. 

Comments:  None. 

Additional  Exception  Handling  Benchmarks: 

1.  Exception  handling  times  may  degrade  due  to  additional  tasks  present  in  the 
system.  Benchmarks  are  needed  that  measure  exception  handling  timings  when 
multiple  tasks  are  present  in  the  system. 

Benchmark:  Measure  the  effect  of  additional  tasks  in  the  system  on  exception 
handling  times  for  all  the  exception  handling  benchmarks  in  the  ACEC  suite. 
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2.  In  many  real-time  systems,  it  is  quite  possible  that  intermediate  operations 
during  the  calculation  of  a  larger  expression  may  exceed  the  system  defined 
limits,  although  the  final  result  may  still  be  within  bounds.  Some 
implementations  may  raise  an  exception  if  the  intermediate  expression  exceeds 
system  defined  limits. 

Benchmark:  Determine  if  an  implementation  raises  NUMERIC  ERR  OR  on  an 
intermediate  operation  when  the  larger  expression  can  be  correctly  computed. 

4.4  Input/Output 

Input/Output  benchmarks  can  be  divided  into  the  following  categories: 

•  TEXT  IO  benchmarks 

•  DIRECTIO  benchmarks 

•  SEQUENTIALIO  benchmarks 

•  Asynchronous  I/O  benchmarks 

4.4.1  TEXTIO 


Table  16  lists  the  ACEC  TEXT  IO  benchmarks  along  with  the  results  of  running 
these  benchmarks  on  the  HP  and  Verdix  Ada  compilers. 

Comments: 

1.  Additional  benchmarks  for  TEXT  IO  are  not  needed. 

4.4.2  DIRECT  IO 


Table  17  lists  the  ACEC  DIRECT  IO  benchmarks  along  with  the  results  of  running 
these  benchmarks  on  the  HP  and  Verdix  Ada  compilers. 

Comments: 

1.  Additional  benchmarks  for  DIRECT_IO  are  not  needed. 

4.4.3  SEQUENTIAL  IO 

Table  18  lists  the  ACEC  SEQUENTIALJO  benchmarks  along  with  the  results  of 
running  these  benchmarks  on  the  HP  and  Verdix  Ada  compilers. 


Comments: 
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1.  Additional  benchmarks  for  SEQUENTIAL_IO  are  not  needed. 

4.4.4  Asynchronous  I/O 

One  of  the  benefits  of  Ada’s  tasking  techniques  is  the  ability  to  implement  true 
asynchronous  I/O.  By  using  Ada  tasks  to  drive  I/O  controllers,  only  the  task  that 
requested  the  I/O  must  wait  for  completion  before  resuming  execution,  while  other 
tasks  within  the  application  program  can  continue  execution  while  I/O  is  being 
processed. 

I/O  blocking  may  not  be  tolerated  in  many  systems.  It  effectively  causes  the  entire 
Ada  program  to  stop  while  an  I/O  is  serviced.  The  effect  is  clearly  most  evident  for 
interactive  input  but,  for  mission  critical  systems  even  physical  disk  I/O  will  cause 
unacceptable  delays  in  the  overall  processing. 

Comments: 

1.  The  ACEC  suite  does  not  have  benchmarks  that  address  asynchronous  I/O. 

Additional  Asynchronous  I/O  Benchmarks 

1.  I/O  blocking  to  devices  other  than  where  clear  delays  are  possible  (such  as  a 
terminal  or  mailbox)  can  be  very  difficult  to  determine.  In  principle  it  is  only 
necessaiy  for  non-blocking  I/O  to  occur  for  physical  I/O  but  when  this  actually 
happens  is  difficult  to  predict  in  many  systems  where  complex  device  caching 
and  buffering  is  automatically  performed.  The  tests  need  to  be  performed  for 
the  following  device  types:  interactive  terminal  and  disk.  For  each  facility  and 
each  of  SEQUENTIAL  IO,  DIRECT  IO,  and  TEXT  IO  the  presence  of 
system-wide  blocking  during  prolonged  processing  should  be  recorded. 

Benchmark:  Blocking  on  READ,  GET,  WRITE,  PUT,  CREATE,  OPEN, 
RESET,  CLOSE,  and  DELETE. 

4.5  Clock  Function 

For  programming  real-time  systems,  the  CLOCK  function  in  the  package 
CALENDAR  is  used  extensively.  Table  19  lists  the  ACEC  CLOCK  function  tests 
along  with  the  results  of  running  these  benchmarks  on  the  HP  and  Verdix  Ada 
compilers. 

Comments: 


1.  These  tests  are  sufficient  to  test  the  CLOCK  function  overhead. 


4.6  Scheduling,  Preemption  and  Delay  Statement 


To  allow  execution  to  switch  among  tasks,  the  scheduler  provided  by  the  runtime 
system  is  entered  at  certain  synchronization  points  in  a  program,  and  the  scheduler 
decides  at  this  point  which  task  has  to  be  executed.  According  to  the  LRM,  an 
implementation  is  free  to  choose  among  tasks  of  equal  priority  or  among  tasks  whose 
priority  has  not  been  defined.  The  minimum  synchronization  (a  implementation  may 
choose  to  have  more)  points  at  which  the  scheduler  is  invoked  are  the  beginning  and 
end  of  task  activations  and  rendezvous.  The  pragma  priority  enables  real-time 
embedded  systems  programmers  to  specify  a  higher  priority  for  more  important  tasks. 
The  priority  is  fixed  at  compile  time  (assuming  that  pragma  priority  is  implemented). 
Hence,  whenever  a  scheduling  decision  has  to  be  made,  the  highest  priority  task 
receives  control  (task  priorities  are  discussed  in  Section  4.1.5). 

Table  20  lists  the  ACEC  benchmarks  along  with  the  results  of  running  these 
benchmarks  on  the  HP  and  Verdix  Ada  compilers. 

Comments:  Observations  about  these  benchmarks  are: 

1.  As  mentioned  in  previous  sections,  benchmarks  that  measure  timing  for  delay 
statement  with  negative  (ss459)  or  0.0  (delayl,  delay8)  argument  do  not  serve 
any  useful  purpose  for  real-time  systems. 

Additional  Scheduling,  Preemption  and  Delay  Statement  Benchmarks: 

Task  preemption  feature  is  very  important  for  real-time  systems.  Preemption  occurs 
for  a  variety  of  reasons  each  of  which  must  be  established.  linked  with  preemption  is 
the  scheduling  algorithm  used  to  determine  which,  of  any  number  of  candidates,  can 
be  processed.  The  test  should  determine,  as  far  as  possible,  the  conditions  under 
which  tasks  are  scheduled  and,  in  particular,  the  order  chosen  where  valid  alternatives 
exist. 

1.  Expiration  of  delay  statement  may  cause  scheduling  to  take  place  and  preempt 
the  running  task. 

Benchmark:  Test  whether  delay  expiration  causes  task  preemption. 


2.  Some  implementations  may  cause  scheduling  decisions  to  take  place  upon  I/O 
completion.  It  should  be  ascertained  if  RTS  system  calls  (such  as  opening  of 
files,  task  creation,  and  rendezvous  handling)  are  themselves  preemptible.  This 
is  of  great  importance  when  dealing  with  multi-priority  systems  and  especially 
where  interrupts  are  possible. 

Benchmark:  Establish  whether  I  JO  completion  causes  task  preemption. 
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3.  An  external  interrupt  may  also  cause  preemption. 

Benchmark:  Establish  whether  external  interrupts  preempt  running  tasks. 


4.7  Chapter  13  Benchmarks 

Chapter  13  benchmarks  are  divided  into  the  following  categories: 

1.  Pragma  PACK:  This  is  considered  here  (as  opposed  to  the  section  that  deals 
with  Pragmas)  as  the  extent  of  packing  performed  by  a  compiler  can  be 
compared  to  other  benchmarks  that  use  the  SIZE  specification  clause. 

2.  SIZE  specification  benchmarks 

3.  Record  repre*  .it  .don  clause  benchmarks 

4.  Attribute  be  chmarks 

5.  UncheckedConversion  benchmarks 

4.7.1  Pragma  Pack 

These  set  of  benchmarks  test  the  packing  capabilities  of  a  Ada  compiler  system  by 
specifying  Pragma  Pack  for  various  objects.  These  tests  measure  both  time  and  space 
utilization.  Some  packing  methods  allocate  a  component  so  that  it  will  span  a  storage 
unit  boundary  while  some  pack  as  densely  as  possible.  The  time  to  access  a 
component  which  spans  a  storage  unit  is  usually  greater  than  when  the  component 
does  not  span  a  boundary.  In  addition  to  measuring  the  time  in  accessing  packed 
objects,  these  test  problems  use  the  representation  attribute  X’SIZE  to  determine  the 
actual  bit  size  of  the  objects  and  compare  this  with  the  predetermined  minimum 
possible  bit  size  for  the  object.  This  shows  the  degree  of  packing  performed  by  the 
system  under  test. 

Tables  21, 22,  and  23  list  the  ACEC  Pragma  PACK  benchmarks  along  with  the  results 
of  running  these  benchmarks  on  the  HP  and  Verdix  Ada  compilers. 

Comments: 

1.  Additional  benchmarks  are  not  needed  for  Pragma  PACK 

4.7.2  Length  Clause:  SIZE  Specification  Benchmarks 

Tables  24  and  25  list  the  ACEC  Length  Clause  SIZE  specification  benchmarks  along 
with  the  results  of  running  these  benchmarks  on  the  HP  and  Verdix  Ada  compilers. 
In  these  test  cases,  an  integer  type  is  declared  and  then  the  TYPE’SIZE 
representation  clause  is  used  to  determine  the  size  in  bits  an  object  of  that  type  can 
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occupy.  An  array  is  declared  of  that  integer  type.  Tests  are  then  performed  on 
components  of  that  array. 

Comments: 

1.  Additional  benchmarks  are  not  needed  for  this  feature. 

4.7 .3  Record  Representation  Clause  Benchmarks 

In  these  tests,  the  record  representation  clause  is  used  to  specify  the  layout  of  a  record 
whose  components  are  boolean  variables  (that  have  the  SIZE  representation  clause 
specified)  as  well  as  a  packed  boolean  array. 

Table  26  lists  the  ACEC  Record  Representation  Clause  benchmarks  along  with  the 
results  of  running  these  benchmarks  on  the  HP  and  Verdix  Ada  compilers. 

Comments: 

1.  Additional  benchmarks  are  not  needed  for  this  feature. 

4.7.4  Attribute  Tests 

These  tests  determine  if  certain  attributes  have  been  implemented.  Table  27  lists  the 
ACEC  Attribute  tests  along  with  the  results  of  running  these  benchmarks  on  the  HP 
and  Verdix  Ada  compilers.  The  attributes  tested  include: 

1.  Test  ADDRESS  attribute  of  a  subroutine,  local  object,  and  dynamic  object. 

2.  Test  SIZE  attribute  of  a  local  object,  and  dynamic  object. 

3.  Test  POSITION,  FIRST  BIT,  and  LAST  BIT  attribute  for  a  record 
component. 

4.  Test  STORAGE  TYPE  attribute  for  an  access  type  and  task  type. 

Comments: 

1.  Additional  benchmarks  are  not  needed  for  this  feature. 

4.7.5  Unchecked_Conversion 

In  real-time  systems,  it  is  very  frequently  required  to  do  a  unchecked_conversion  from 
one  type  to  another.  These  set  of  benchmarks  test  the  time  required  to  do  a 
unchecked_conversion  from  one  type  to  another.  Table  28  lists  the 
unchecked_conversion  benchmarks  along  with  the  results  of  running  these 
benchmarks  on  the  HP  and  Verdix  Ada  compilers. 
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Comments:  None. 

Additional  Unchecked  Conversion  Benchmarks: 

1.  In  many  real-time  systems  a  string  object  may  be  converted  to  an  integer  via 
unchecked  conversion  as  well  as  an  array  of  floats  may  be  converted  to  a  record 
of  floats. 

Benchmark :  Measure  the  time  for  UNCHECKED  JCONVERSION  to  move  a 
STRING  object  to  another  INTEGER  object. 


2.  Benchmark:  Measure  the  time  to  do  an  unchecked  conversion  of  an  array  of  10 
floating  components  into  a  record  of  10  floating  components. 

4.8  Interrupt  Handling 

In  real-time  embedded  systems,  efficient  handling  of  interrupts  is  very  important. 
Interrupts  are  critical  to  the  ability  of  the  system  to  respond  to  real-time  events  and 
perform  its  required  functions  and  it  is  essential  that  the  system  responds  to  the 
interrupt  in  some  fixed  amount  of  time. 

Table  29  lists  the  ACEC  Interrupt  handling  benchmarks. 

Comments:  None. 

Additional  Interrupt  Handling  Benchmarks: 

1.  In  many  real-time  systems,  it  is  important  that  interrupts  are  not  lost  when  an 
interrupt  is  being  handled  and  another  interrupt  is  received  from  the  same 
device. 

Benchmark:  Determine  if  an  interrupt  is  lost  when  an  interrupt  is  being  handled 
and  another  interrupt  is  received  from  the  same  device. 

2.  An  implementation  may  cause  scheduling  decisions  on  receipt  of  an  interrupt. 
This  may  not  be  desirable  in  some  real-time  systems. 

Benchmark:  Determine  if  an  interrupt  entry  call  invokes  any  scheduling  decisions. 

3.  The  handler  deals  with  high  priority  interrupts,  and  is  therefore  allocated  a  high 
task  priority.  However,  it  can  be  interrupted  outside  the  rendezvous  by  a  low 
priority  interrupt  and  cannot  guarantee  to  return  to  the  accept  statement  in  time 
to  catch  the  next  high  priority  interrupt 

Benchmark:  Determine  if  accept  statement  executes  at  the  priority  of  the  hardware 


-32- 


interrupt,  and  if  priority  is  reduced  once  a  synchronization  point  is  reached 
following  the  completion  of  accept  statement. 

4.9  Pragmas 

There  are  certain  predefined  pragmas  which  are  expected  to  have  an  impact  on  the 
execution  time  and  space  of  a  program.  These  include:  SUPPRESS,  OPTIMIZE, 
SHARED,  INLINE,  PACK,  CONTROLLED,  and  PRIORITY.  Benchmarks  for 
Pragma  INLINE  are  covered  under  Subprogram  Overhead  in  Section  4.10.  Pragma 
PACK  is  covered  under  Chapter  13  benchmarks  (Section  4.7),  Pragma 
CONTROLLED  is  covered  under  Memory  Management  benchmarks  (Section  4.2), 
and  Pragma  PRIORITY  is  covered  under  Tasking  (Section  4.1). 

4.9.1  Pragma  SUPPRESS 

The  benchmarks  for  pragma  SUPPRESS  determine  the  improvement  in  execution 
time  when  pragma  SUPPRESS  is  used  Pragma  SUPPRESS  causes  the  compiler  to 
omit  the  corresponding  exception  checking  (RANGECHECK,  STORAGE  CHECK 
etc.)  that  occurs  at  runtime. 

Table  30  lists  the  ACEC  Pragma  SUPPRESS  benchmarks  along  with  the  results  of 
running  these  benchmarks  on  the  HP  and  Verdix  Ada  compilers. 

Comments:  None. 

Additional  Pragma  SUPPRESS  Benchmarks: 

1.  Timing  has  also  to  be  measured  for  other  kinds  of  checks  using  Pragma 
SUPPRESS. 

Benchmark :  Pragma  SUPPRESS  is  used  for  these  checks:  Access  JCheck 
Index JCheck  Length  JCheck  Storage  _Check  and  Elaboration  JCheck 

4.9.2  Pragma  OPTIMIZE 

The  benchmarks  for  pragma  OPTIMIZE  determine  the  improvement  in  execution 
time  when  the  pragma  is  used. 

Comments: 

1.  ACEC  has  no  pragma  OPTIMIZE  benchmarks. 

Additional  Pragma  OPTIMIZE  Benchmarks: 
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1.  Timing  has  to  be  measured  for  improvement  in  execution  time  pragma 
OPTIMIZE  is  used  with  options  TIME  and  SPACE. 

Benchmark :  Determine  improvements  in  execution  time  when  pragma  OPTIMIZE 
is  used  with  options  TIME  and  SPACE. 

4.9.3  Pragma  SHARED  Benchmarks 

With  multiple  tasks  executing,  there  may  be  an  instance  where  the  same  nonlocal 
variable  must  be  accessed.  Pragma  SHARED  is  the  mechanism  that  designates  that  a 
variable  is  shared  by  two  or  more  tasks.  Pragma  SHARED  directs  the  RTE  to 
perform  updates  of  the  shared  variable  copies  each  time  they  are  updated,  but  the 
overhead  may  be  significant. 

Comments: 

1.  There  are  no  ACEC  Pragma  SHARED  benchmarks. 

Additional  Pragma  SHARED  Benchmarks: 

1.  The  overhead  involved  in  updating  a  shared  integer  variable  is  compared  to  the 
overhead  involved  in  updating  an  integer  variable  that  is  not  shared. 

Benchmark :  Determine  the  overhead  due  to  Pragma  SHARED  when  two  tasks 
access  a  shared  integer  variable. 

The  main  program  updates  a  shared  integer  variable.  This  integer  variable  is 
also  updated  by  another  task. 

2.  The  overhead  involved  in  updating  a  shared  integer  variable  during  a  rendezvous 
is  compared  to  the  overhead  involved  in  updating  an  integer  variable  (that  is  not 
shared)  during  a  rendezvous. 

Benchmark:  Determine  the  overhead  in  rendezvous  time  when  a  shared  variable  is 
updated  during  the  rendezvous. 

4.10  Subprogram  Overhead 

In  Ada,  subprograms  rank  high  among  program  units  from  a  system  structure  point  of 
view.  If  the  subprogram  overhead  is  high,  then  the  compiler  can  generate  INLINE 
expansion  at  the  cost  of  increasing  the  size  of  the  object  code.  However,  if  calls  to 
that  subprogram  are  made  from  a  lot  of  places,  then  the  pragma  INLINE  defeats  the 
purpose  due  to  increase  in  size  of  object  code. 

Tables  31  and  32  list  the  ACEC  Subprogram  overhead  tests  along  with  the  results  of 
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running  these  benchmarks  on  the  HP  and  Verdix  Ada  compilers. 

Comments:  None. 

Additional  Subprogram  Overhead  Tests: 

1.  Subprogram  overhead  timings  have  to  be  measured  when  various  parameters 
are  passed  during  a  procedure  call. 

Benchmark:  Various  numbers  of  parameters  of  types  INTEGER  and 
ENTJMERATION  are  passed  to  determine  the  subprogram  overhead  associated 
with  simple  parameter  passing.  Then  composite  objects  (arrays  and  records )  are 
passed  to  determine  if  they  are  passed  by  copy  or  reference.  Finally,  the 
subprogram  is  called  with  formal  parameters  of  an  unconstrained  composite  type. 

All  of  the  tests  include  passing  the  parameters  with  modes  in,  out,  and  in  out. 
All  of  the  tests  involve  two  different  types  of  subprogram  calls,  one  to  a 
subprogram  that  is  a  part  of  the  same  package  as  the  caller,  and  the  other  to  a 
subprogram  in  a  package  other  than  the  one  in  which  the  caller  resides. 

2.  Benchmarks  for  subprogram  overhead  that  involve  the  use  of  package 
instantiations  of  generic  code  are  also  needed. 

Benchmark:  All  of  the  above  tests  for  both  inter-package  and  intra-package 
procedure  calls  are  repeated  with  the  subprograms  being  part  of  a  generic  unit 

4.11  Numeric  Computation 

An  embedded  system  must  be  able  to  represent  real-world  entities  and  quantities  to 
perform  related  manipulations  and  computations.  There  should  be  support  for 
numerical  computation,  units  of  measure  (including  time),  and  calculations  and 
formula  from  physics,  chemistry  etc.  Numeric  computation  benchmarks  are  discussed 
under  two  separate  logical  headings:  a)  Mathematical  Computation  Benchmarks  and 
b)  Benchmarks  for  Arithmetic  on  Type  TIME  and  DURATION.  The  ACEC  suite 
has  over  300  numeric  computation  benchmarks  which  belong  to  different  categories. 
Hence,  these  benchmarks  have  not  been  presented  in  the  form  of  tables. 

4.11.1  Mathematical  Computation  Benchmarks 

These  benchmarks  range  from  simple  integer  additions  to  calculation  of  complex 
mathematical  equations.  ACEC  has  the  following  categories  of  mathematical 
computation  benchmarks: 

1.  Floating  point  addition,  multiplication,  division 
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2.  Integer  addition,  division,  multiplication 

3.  Natural  Integer  addition,  division,  multiplication,  mod,  rem 

4.  Fixed  point  addition,  multiplication,  division 

5.  Type  conversions  from  integer  to  real,  floating  point  literal  to  integer,  integer  to 
float,  convert  one  fixed  point  to  another,  convert  double  to  real,  convert  real  to 
double 

6.  Integer  addition,  division 

7.  exp,  In,  sin,  cos,  abs,  sqrt,  atan,  sgn 

8.  mod,  and  rem  operators 

9.  polynomial  evaluation 

10.  Long  integer  assignment,  addition,  subtraction,  multiplication,  division,  rem, 
conversion  from  integer  to  long  integer 

11.  extended  precision  floating  point  assignment,  addition,  division,  abs,  sin,  cos, 
exp,  In,  sqrt,  atan,  extended  precision  floating  point  array  assignment 

Comments:  . 

1.  As  far  as  mathematical  computation  benchmarks  are  concerned,  the  ACEC  set 
is  comprehensive  and  complete. 

4.11.2  Benchmarks  For  Arithmetic  On  Type  TIME  and  DURATION 

For  real-time  embedded  systems,  it  is  necessary  to  dynamically  compute  values  of  type 
TIME  and  DURATION.  An  example  of  such  a  computation  is  the  difference 
between  a  call  to  the  CLOCK  function  and  a  calculated  TIME  value.  This  value  may 
be  used  as  a  parameter  in  the  delay  statement.  If  the  overhead  involved  in  this 
computation  is  significant,  the  actual  delay  experienced  will  be  longer  than  anticipated 
which  could  be  critical  for  real-time  systems. 

ACEC  benchmarks  for  arithmetic  on  type  TIME  and  DURATION  are  divided  in  the 
following  categories: 

1.  Addition  of  variables  of  type  CALENDAR.TTME. 

2.  Comparison  of  variables  of  type  CALENDAR.TTME. 

3.  Comparison  of  type  DURATION  with  SECONDS(TTME). 

4.  Call  on  CALENDAR.TTME  OF  function. 

Comments:  None. 

Additional  Benchmarks  For  Arithmetic  On  Type  TIME  and  DURATION: 
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1.  Many  real-time  systems  may  have  the  need  to  compute  values  using  the "+"  and 
functions  provided  in  the  package  calendar. 

Benchmark:  Measure  the  overhead  associated  with  a  call  to  and  return  from  the 
"+ "  and  "-"functions  provided  in  the  package  CALENDAR. 

Times  are  measured  for  computations  involving  just  variables  and  both 
constants  and  variables  of  Type  TIME  and  DURATION.  The  variables  have 
predefined  values.  Although  both  "+'*  functions  are  essentially  the  same  (only 
the  order  of  parameters  reversed)  both  are  tested.  This  is  done  because  a 
discrepancy  in  the  time  needed  to  complete  the  computation  will  occur  if  one  of 
the  functions  is  implemented  as  a  call  to  the  other. 

4.12  Real-Time  Paradigms 

Users,  system  programmers,  and  academicians  have  found  a  number  of  useful 
paradigms  for  building  concurrency.  These  real-time  paradigms  can  be  coded  in  Ada 
and  benchmarked. 

Table  33  lists  the  ACEC  real-time  paradigms  along  with  the  results  of  running  these 
benchmarks  on  the  HP  and  Verdix  Ada  compilers. 

Comments:  None. 

Additional  Real-time  Paradigms: 

1.  Many  real-time  implementations  require  buffered  and  unsynchronized 
communication  between  tasks.  Due  to  the  rendezvous  being  a  synchronous  and 
unbuffered  message  passing  operation,  intermediary  tasks  are  needed  to 
uncouple  the  task  interaction  to  allow  tasks  more  independence  and  increase  the 
amount  of  concurrency.  Various  combinations  of  intermediary  tasks  are  used  in 
different  task  paradigms  to  create  varying  degrees  of  asynchronism  between  a 
producer  and  consumer.  The  benchmarks  defined  here  evaluate  the  cost  of 
introducing  intermediary  tasks  for  various  real-time  tasking  paradigms.  The 
goal  of  these  benchmarks  is  to  give  real-time  programmers  a  feel  for  the  cost  of 
using  such  paradigms  in  a  real-time  embedded  application  and  to  avoid  using 
such  paradigms  if  the  cost  is  unacceptable  for  a  real-time  system. 

Benchmark:  Measure  the  cost  of  rendezvous  between  a  producer  and  consumer. 
The  task  that  is  the  source  of  the  information  is  called  the  producer  and  the  task 
that  is  the  recipient  of  the  information  is  called  the  consumer. 

Benchmark:  Measure  the  cost  of  rendezvous  using  buffer  tasks.  A  buffer  is  pure 
server  task  that  provides  for  one  entry  for  storing  of  items  in  a  buffer  and  another 
entry  for  providing  items  from  the  buffer.  Both  the  consumer  and  the  producer  call 
the  buffer  task  to  obtain  a  piece  of  information. 
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Benchmark :  Measure  the  cost  of  rendezvous  using  a  buffer  and  and  transporter. 
This  scheme  uses  two  intermediary  tasks  between  the  producer  and  the  consumer. 

Benchmark :  Measure  the  cost  of  rendezvous  using  a  buffer  and  and  two 
transporters.  If  both  the  producer  and  consumer  wish  to  communicate  via  a  buffer 
and  both  need  to  be  called  tasks,  it  is  necessary  to  use  a  transporter  on  each  side  of 
the  buffer.  This  results  in  the  producer-transporter-buffer-transporter-consumer 
paradigm. 


2.  A  monitor  is  commonly  used  for  controlling  a  system’s  resources.  For  example, 
read  and  write  operations  to  a  disk  are  usually  controlled  by  a  monitor  that 
ensures  the  integrity  of  data  on  the  disk.  This  is  also  known  as  mutual  exclusion. 
Monitors  can  be  implemented  to  have  controlled  access  to  a  shared  data  pool 
Monitors  can  be  implemented  via  semaphores,  event  signaling,  and  rendezvous 
mechanism.  The  implementation  via  semaphores  and  event  signaling  is 
essentially  the  same. 

Benchmark:  Measure  time  to  access  data  in  a  pool  using  a  monitor. 

Any  number  of  processes  are  allowed  to  read  the  pool  simultaneously,  but  no 
reads  are  permitted  during  a  write  operation.  The  monitor  developed  is  used  to 
control  the  reading  and  writing  of  data  to  the  pool.  Two  implementations  of  the 
monitor  can  be  considered:  the  first  using  semaphores,  and  the  second  using  the 
Ada  rendezvous  mechanism. 

4.13  Composite  Benchmarks 

A  composite  benchmark  is  defined  as  a  program,  within  the  context  of  the  application 
domain,  that  looks  at  the  interaction  between  Ada  features  rather  than  the 
performance  of  individual  features  themselves.  The  purpose  of  running  a  composite 
benchmark  is  to  aid  in  the  selection  of  a  suitable  compiler  and  runtime  for  a  particular 
application. 

The  ACEC  also  includes  examples  from  actual  application  code.  This  code  contains 
test  problems  representative  of  how  Ada  is  being  used  in  practice.  These  are  example 
test  problems  drawn  from  Ada  programs  extracted  from  projects.  Table  34  lists  those 
benchmarks  along  with  the  results  of  running  these  benchmarks  on  the  HP  and  Verdix 
Ada  compilers. 

Comments:  Observations  about  the  ACEC  composite  benchmarks  are: 

1.  The  code  contained  in  the  composite  benchmarks  is  not  very  useful  and  does  not 
provide  any  relevant  information  for  compiler  evaluation  for  real-time  systems. 


Additional  Composite  Benchmarks: 
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1.  Composite  benchmarks  can  be  developed  for  a  number  of  individual 
applications  such  as  Intelligence/Electronic  Warfare  (IEW)  systems,  avionics 
systems,  etc.  More  information  about  the  process  of  development  of  composite 
benchmarks  is  contained  in  a  report  titled  "Ada  Composite  Benchmark  for 
Intelligence/Electronic  Warfare  Systems"  [11]. 

Benchmark:  Develop  composite  benchmarks  for  IEW  systems  [11]. 

4.14  Portability  Of  the  ACEC  Benchmarks 

The  ACEC  tests  essentially  consist  of  two  sets  of  tests: 

a.  Tests  that  do  not  depend  on  the  MATH  packages  supplied  with  the  ACEC  (e.g. 
the  tasking  tests) 

b.  Tests  that  depend  on  the  MATH  packages. 

The  tests  that  do  not  depend  upon  the  MATH  packages  can  easily  be  ported  from  one 
Ada  compiler  to  another.  More  effort  is  required  to  port  tests  that  depend  upon  the 
MATH  packages.  This  section  describes  the  steps  that  have  to  be  performed  to  port 
the  ACEC  to  any  Ada  compiler  configuration.  The  ACEC  is  intended  to  run  on  bare 
targets  as  well  as  targets  with  operating  systems. 

4.14.1  Modification  To  the  Command  Files 

The  first  step  is  to  modify  the  command  files  that  run  the  ACEC  benchmarks.  The 
command  files  cmp.unx,  cmp  tst.unx  and  cmpbase.unx  that  run  the  ACEC 
benchmarks  have  to  be  modified  to  reflect  the  compilation  and  linking  commands  of 
the  Ada  compiler  system. 

Porting  Effort:  Easily  portable. 

4.14.2  Modification  To  the  Base  ACEC  Files 

The  second  step  involves  modification  to  the  a  set  of  files  known  as  the  base  ACEC 
files.  These  are  globaLada,  math_dep.ada,  math_test.ada,  and  dbl_mathtest.ada. 

1.  globaLada: 

An  ACEC  user  can  choose  to  run  the  timing  loop  using  CPU  time  rather  than 
elapsed  time.  Using  CPU  time  permits  the  collection  of  measurements  on 
multiprogramming  target  systems  without  having  to  shut  the  system  down  to 
eliminate  contending  jobs.  The  ACEC  benchmarks  utilize  the  function 
CPU_TIME_CLOCK  which  is  defined  in  the  package  GLOBAL  in  the  file 
globaLada.  This  function  is  implementation  dependent  and  has  to  be  written  for 
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each  Ada  compiler  system. 

If  the  system  does  not  support  an  integer  type  with  at  least  32  bits  of  precision, 
the  declaration  of  the  type  "BIGINT"  and  "BIGNAT'  will  not  compile  and  must 
be  removed. 

If  the  system  does  not  support  a  floating  point  type  with  9  digits  of  precision,  the 
declaration  of  the  type  "DOUBLE"  will  not  compile  and  must  be  removed. 

The  function  "ADDRESSTOINT'  converts  a  value  of  type 
SYSTEM.  AD  DRESS  to  an  integer  type.  This  function  is  used  to  compute  the 
code  expansion  sizes  by  subtracting  the  address  values  of  two  label’ ADDRESS 
attributes  (or  of  two  type  ADDRESS  variables  obtained  by  a  GETADR 
function).  On  different  systems,  SYSTEM.ADDRESS’SIZE  will  differ  forcing  a 
modification  to  the  return  type  of  this  function. 

Porting  Effort:  Easily  implementable. 


2.  mathdep.ada : 

The  package  MATH  DEPENDENT  in  math  dep.ada  has  to  be  adapted  to 
reflect  both  the  characteristics  of  the  target  machine  floating  point  hardware 
and  the  facilities  which  the  Ada  compilation  system  provides  to  manipulate  bit 
fields  in  floating  point  variables.  The  size  and  location  of  the  sign,  exponent,  and 
mantissa  of  a  floating  point  number  are  critical,  as  are  other  representation 
details  such  as  the  encoding  of  the  exponent  field. 

Porting  Effort:  Non- trivial. 


3.  mathtestada  and  dbljnathtest  ada: 

The  file  mathtest.ada  has  the  program  MATHTEST  which  tests  the  math 
routines.  MATHTEST  requires  a  package  MACHINE  which  contains  some 
hardware  dependent  constants  which  are  modified  for  the  ACEC  benchmarks. 
These  values  are  obtained  from  the  compiler  documentation. 

The  file  dbl_mathtest.ada  has  the  program  DBLMATHTEST  which  tests  the 
double  precision  math  routines.  DBL  MATHTEST  requires  a  package 
DBL  MACHINE  which  contains  some  hardware  dependent  constants  which 
are  modified  for  the  ACEC  benchmarks.  This  information  can  be  obtained  from 
the  documentation  for  the  HP  Ada  compiler  system. 


Porting  Effort:  Non-triviaL 
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4.143  Input/Output 

The  third  step  is  to  make  sure  that  TEXT_IO  and  FLOAT_IO  are  supported  on  the 
target  on  which  the  ACE C  suite  has  to  be  run.  The  ACEC  outputs  strings  containing 
numeric  results  of  performance  tests.  If  TEXTIO  and  FLOATJO  are  not 
supported,  the  results  of  the  tests  cannot  be  displayed  without  modification  to  the 
timing  code  loop.  If  FLOAT  IO  is  not  directly  supported,  users  will  have  to  develop 
work  arounds. 

If  the  Ada  system  supports  a  complete  version  of  the  TEXT_IO  package,  this 
requirement  causes  no  problems.  However,  some  Ada  compilers  targeted  to  real-time 
systems  (where  the  hardware  is  limited)  may  limit  their  I/O  facilities.  The  standard 
timing  output  is  in  microseconds  to  the  nearest  tenth.  What  is  needed  is  the  capability 
to  output  real  numbers.  If  thirty-two  bit  integers  are  available,  one  viable  option  is  to 
multiply  timing  results  by  10  and  then  output  INTEGR’IMAGE.  A  user  may  have  to 
write  a  floating  point  to  a  text-string  conversion  routine. 

For  bare  machine  implementations  of  Ada,  the  effort  required  to  get  TEXT  IO  to 
work  well  enough  to  output  results  of  the  timing  and  sizing  measurement  on  a  console 
can  be  large.  Portions  of  the  Ada  runtime  library  may  need  to  be  modified,  I/O  device 
drivers  may  need  to  be  written  and  tested 


Porting  Effort:  Non-trivial. 
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5.  Conclusions 


Benchmarking  Ada  implementations  to  determine  their  suitability  for  real-time 
systems  is  an  extremely  complex  task.  This  job  is  made  even  more  difficult  due  to 
differing  requirements  of  various  real-time  applications.  The  ACEC  benchmarks 
provide  a  good  start  for  benchmarking  Ada  compilers  meant  for  real-time 
applications.  However,  the  ACEC  benchmarks  need  to  be  augmented  with  more 
benchmarks  in  certain  areas  as  outlined  in  this  report.  It  is  hoped  that  the  results  of 
this  study  will  enable  appropriate  extensions  to  the  ACEC  benchmark  suite  so  that 
they  are  more  useful  in  benchmarking  Ada  compilers  meant  for  real-time  systems. 
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Appendix  A:  ACEC  Real-time  Benchmarks  and  Execution  Results 


Appendix  A  lists  the  ACEC  real-time  benchmarks.  It  also  presents  the  results  of 
running  the  ACEC  real-time  benchmarks  on  the  following  Ada  compilers: 

•  HP-Ada  Compiler  (Releases  3.25  and  4.35)  running  on  HP  9000/350  machine 
under  HP-UX  Release  6.2. 

•  Verdix  Ada  Compiler  (Release  5.41)  hosted  on  a  Sun  3/60  and  targeted  to  a 
Motorola  68020  bare  machine. 

The  hardware  and  software  configurations  for  the  two  compilers  is  as  follows: 

HP  Testbed  Hardware  and  Software 

The  hardware  used  for  benchmarking  was  Hewlett-Packard  9000/350  CPU  running 
HP-UX  V  6.2.  The  set  p  can  be  summarized  as  follows: 

Host:  HP  9000/350  running  HP-UX  V  6.2. 

Compiler:  Self-hosted  HP  (basically  the  Alsys  Ada  Compiler) 

Ada  Development  System  Version  3.25  and  4.35. 

Target:  Same  as  the  host. 

Verdix  Testbed  Hardware  and  Software 

The  hardware  used  for  benchmarking  was  Sun  3/60  CPU  running  Sun  Unix  4.2 
Release  3.5,  linked  to  a  single  12.5  Mhz  Motorola  68020  single  board  computer 
enclosed  in  a  multibus  chasis.  The  setup  can  be  summarized  as  follows: 


Host:  Sun  3/60,  running  Sun  Unix  4.2  Release  3.5 

Compiler:  Verdix  Ada  Development  System  targeted  to  Motorola  MC68020 

targets,  release  5.41 

Target:  GPC68020  (based  on  Motorola  MC68020  microprocessor) 

multibus-compatible  computer  board  having  12.5  Mhz 
MC68020  microprocessor,  a  MC68881  floating  point 
co-processor,  and  2  megabyte  of  RAM. 


TTie  interrupt  handling  benchmarks  were  not  executed  due  to  lack  of  the  required 
hardware  to  produce  external  interrupts.  A  comprehensive  performance  evaluation  of 
the  HP  Ada  compilers  obtained  by  running  the  ACEC  and  Real-time  benchmarks  is 
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presented  in  reference  [10].  For  the  Verdix  compiler,  comparing  the  ACEC  real-time 
benchmark  results  to  the  Real-time  benchmarks  (listed  in  [4])  shows  that  the  two  sets 
of  results  are  in  sync  for  the  common  features  measured  by  those  benchmarks. 


TABLE  3.  Measure  Time  For  Simple  Rendezvous 


TABLE  4.  Rendezvous  Performance  With  Varying  Number  of  Tasks 


TABLE  6.  Rendezvous  Calls  with  Conditional  Selects 


TABLE  7.  Selective  Wait 


TABLE  9.  Abort  Statement 


TABLE  10.  Miscellaneous  Tasking  Benchmarks 


TABLE  11.  Habermann-Nassi  Tasking  Optimizations 


TABLE  12.  Other  Tasking  Optimizations 


TABLE  13.  Memory  Allocation  Timing  Benchmarks 


TABLE  14.  Memory  Allocation/Deallocation  Benchmarks 
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TABLE  IS.  Exception  Handling 
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is  raised  during  a  nested  function  call.  A  poor  implementation  may  leave 

garbage  on  the  stack  associated  with  function  call  and  actual  parameters.  4763.0  3302.6 


TABLE  16.  TEXT  IO  Benchmarks 


TABLE  17.  DIRECT  IO  Benchmarks 


TABLE  19.  CLOCK  Function 


TABLE  20.  Scheduling  and  Delay  Statement 


TABLE  21.  Pragma  PACK 
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TABLE  22.  Pragma  PACK  (Continued) 


ime  for  both  spanning  and  nonspanning  accesses,  bit  size  16  508.1  337.7  617.7 

'ime  for  allocating  a  unpacked  array  component  to  a  packed  array  component,  bit  size  3  116.2  226.9  Error 


TABLE  23.  Pragma  PACK  (Continued) 
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of  bits  and  OR  operator  20.8  4.1  363 


TABLE  24.  SIZE  Representation  Clause  Benchmarks 


TABLE  25.  SIZE  Representation  Clause  Benchmarks  (Continued) 
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TABLE  28.  Unchecked  Conversion 


TABLE  31.  Subprogram  Overhead  Tests 


TABLE  32.  Subprogram  Overhead  Tests  (Continued) 


TABLE  34.  Composite  Benchmarks 
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Appendix  B:  ACEC  Benchmarks  From  a  Real-time  Perspective 


The  ACEC  benchmarking  suite  was  primarily  developed  for  evaluating  the  runtime 
performance  of  Ada  compilers.  Since  the  suite  is  comparatively  new,  not  man/ 
organizations  have  run  the  ACEC  with  the  intent  of  evaluating  compilers.  Being  one 
of  the  first  organizations  outside  of  the  Air  Force  that  evaluated  and  ran  the  ACEC  on 
different  compiler  implementations,  a  number  of  problems  about  the  ACEC  have 
surfaced.  These  problems  are  discussed  below. 

1.  INCLUDE  Preprocessor  The  ACEC  timing  code  loop  consists  of  four  (4)  code 
files  which  are  incorporated  into  the  source  by  a  preprocessor  (INCLUDE) 
which  supports  text  inclusion.  INCLUDE  is  written  as  an  Ada  program  and  has 
to  be  modified  on  systems  which  do  not  support  the  concept  of  file  suffixes  as 
well  as  Ada  compilers  which  require  Ada  programs  to  have  a  suffix  ".a"  instead 
of  ".ada". 

However,  the  biggest  problem  with  INCLUDE  is  that  for  cross-compilers  the 
INCLUDE  command  runs  on  the  target  and  for  bare  machine  implementations 
of  Ada  that  may  not  support  a  file  system,  the  INCLUDE  preprocessor  will  not 
work.  However,  if  the  target  does  support  a  file  system,  it  takes  a  very  long  time 
for  a  single  ACEC  test  to  run  through  the  INCLUDE  preprocessor  (on  the 
Verdix  cross  compiler,  it  took  nearly  30  minutes  to  run  a  file  through  the 
INCLUDE  preprocessor). 


2.  Large  Number  of  Tests:  The  ACEC  suite  consists  of  over  1000  tests  and  many 
(“25%)  of  these  tests  do  not  provide  any  useful  information  about  compiler 
evaluation.  Also,  to  compile  and  run  the  whole  suite  can  take  somewhere  from  3 
to  5  days.  Once  the  tests  have  been  run,  it  is  extremely  difficult  for  an 
organization  to  select  the  relevant  tests  (from  a  real-time  perspective)  and  then 
interpret  the  results  in  order  to  do  a  thorough  compiler  evaluatioa  Substantial 
effort  may  be  required  on  the  part  of  an  organization  to  run  the  ACEC  and 
interpret  the  results  to  perform  compiler  evaluatioa  An  organization  may  not 
have  the  time  and  resources  to  run  the  ACEC  and  then  interpret  the  results  for 
compiler  evaluatioa 

3.  MEDIAN  Analysis  Tool :  The  primary  focus  of  the  ACEC  is  on  comparing 
performance  data  between  different  compilation  systems  rather  than  on  studying 
the  results  of  one  particular  system.  The  MEDIAN  analysis  tool  isolates 
problems  where  a  system  does  unusually  well  or  poorly.  The  problem  with  this 
kind  of  analysis  is  that  if  for  example  two  compilers  are  tested  and  both 
compilers  have  nearly  similar  rendezvous  timings,  the  timings  may  not  meet  the 
real-time  requirements.  The  MEDIAN  analysis  tool  will  not  isolate  this 
problem,  and  an  organization  will  have  to  analyze  the  individual  test  cases  to 
gain  more  information.  If  an  organization  just  depends  on  the  output  of 
MEDIAN,  it  is  quite  possible  that  problems  with  the  runtime  performance  of 
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real-time  Ada  features  may  not  be  detected.  Also,  the  summary  statistics  attach 
equal  weight  to  all  the  problems  and  although  one  compiler  may  have  a  better 
summary  data,  it  does  not  imply  that  it  is  better  than  the  other  compiler  in  areas 
that  are  of  particular  interest  for  an  application.  Also,  it  is  a  tremendous 
amount  of  effort  to  analyze  the  statistical  numbers  produced  for  all  the 
problems  ( >  1000). 


4.  Ada  Application  Code:  The  ACEC  contains  code  from  some  applications  that 
were  developed  in  organizations  using  Ada.  This  code  is  not  very  useful  and 
does  not  provide  any  relevant  information  for  compiler  evaluation  for  real-time 
systems.  The  code  of  these  composite  benchmarks  is  not  designed  using  good 
software  engineering  practices  and  it  appears  that  Fortran  code  may  have  just 
been  converted  to  Ada.  Also,  there  is  no  information  to  interpret  the  results 
produced  by  these  benchmarks  and  it  is  extremely  difficult  to  go  through  the 
code  and  figure  out  what  the  results  of  the  benchmarks  mean. 


5.  Interpretation  of  Results:  In  many  instances,  it  is  not  clear  as  to  why  a  certain  test 
has  been  provided  and  what  is  the  purpose  of  running  that  test.  Also,  in  many 
cases,  the  benchmark  code  has  to  be  read  in  order  to  interpret  the  results  of  the 
output.  More  information  needs  to  be  provided  in  order  to  help  interpret  the 
results  produced  by  running  the  benchmarks. 

6.  Large  Number  of  Iterations:  The  ACEC  tests  dynamically  compute  the  number 
of  iterations  necessary  to  obtain  measurement  within  a  specified  accuracy.  For 
many  tasking  tests,  it  is  quite  possible  that  the  number  of  iterations  is  rather 
large,  so  that  the  benchmark  could  run  out  of  memory  while  executing  the 
required  number  of  iterations  (due  to  memory  being  allocated  as  more  tasks  are 
activated).  This  was  experienced  in  the  case  of  the  Verdix  Ada  compiler 
targeted  to  the  Motorola  68020  bare  machine  with  2  megabytes  of  RAM. 


